Papers with dynamic mask attention network
Mask Attention Networks: Rethinking and Strengthen Transformer (2021.naacl-main)
Copied to clipboard
Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, Xuanjing Huang
| Challenge: | Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. |
| Approach: | They propose to combine SAN and Feed-Forward Networks to create a dynamic mask attention network with a learnable mask matrix which can model localness adaptively. |
| Outcome: | The proposed model outperforms the original Transformer on translation and text summarization tasks. |